Rear-view multi-functional camera system

ABSTRACT

A rear-view camera system for a vehicle. The rear-view camera system includes a first camera positioned on a rear side of the vehicle. The first camera produces a first data set that includes image data corresponding to a first field of view. A second camera is positioned on the rear side of the vehicle spaced a horizontal distance from the first camera. The second camera produces a second data set that includes image data corresponding to a second field of view. The first field of view and the second field of view cover a space adjacent the rear side of the vehicle. An electronic control unit receives the first data set and the second data set and processes the first data set and the second data set to determine the presence of an object in the space adjacent the rear side of the vehicle. The electronic control unit is configured to stitch the first data set and the second data set to produce a panoramic image that corresponds to the space adjacent the rear side of the vehicle. A human-machine interface includes a viewing screen positioned adjacent an operator of the vehicle, and the human-machine interface is configured to receive the panoramic image and display the panoramic image on the viewing screen.

BACKGROUND

The present invention relates to a rear-view multi-functional camerasystem.

There are various situations in which it is desirable to see or sensebehind or to the rear of an automobile or other vehicle. In mostinstances, the requirements of one vision application will vary from therequirements of other applications. For example, the requirements of arear-vision or rear-view camera system in a passenger vehicle oftendiffer from the requirements of a rear-vision or rear-view camera systemin a commercial truck.

SUMMARY

Due, at least in part, to the different sensing coverage behind thevehicles in different vision applications, system designers havestruggled to find one single sensing configuration that fulfills mostapplications with sufficient data integrity.

The present invention provides a rear-view configuration that covers upto a 180° field of view with high image resolution for viewing purposesbehind the vehicle. It also provides high-integrity depth and motioninformation for multiple functionalities including vehicle comfortsystems and vehicle safety systems. The camera system operates underboth forward and reverse driving conditions.

Vehicle sensing systems used in vehicles can be used to alert anoperator of the vehicle to adverse conditions that exist or arepredicted to occur in the future. Such vehicle sensing systems may alerta driver to the presence of an object behind the vehicle when driving inreverse, calculate the motion of an object, and predict a potentialcollision and time of collision, provide a virtual image of the objectsbehind the vehicle, calculate distances, actively control safetyfeatures in a vehicle, actively control braking action and steeringangle of a vehicle, and perform other functions. The vehicle sensingsystem of the present invention performs multiple functions, eliminatingthe need for multiple sensing systems.

In one embodiment, the invention provides a rear-view camera system fora vehicle. The rear-view camera system includes a first camerapositioned on a rear side of the vehicle. The first camera produces afirst data set that includes image data corresponding to a first fieldof view. A second camera is positioned on the rear side of the vehiclespaced a horizontal distance from the first camera. The second cameraproduces a second data set that includes image data corresponding to asecond field of view. The first field of view and the second field ofview cover a space adjacent the rear side of the vehicle. An electroniccontrol unit receives the first data set and the second data set andprocesses the first data set and the second data set to determine thepresence of an object in the space adjacent the rear side of thevehicle. The electronic control unit is configured to stitch the firstdata set and the second data set to produce a panoramic image thatcorresponds to the space adjacent the rear side of the vehicle.

The camera system includes a human-machine interface (“HMI”) that has aviewing screen positioned adjacent an operator of the vehicle. The HMIis configured to receive the panoramic image and display the panoramicimage on the viewing screen.

In another embodiment, the invention provides a method of detecting anobject adjacent a rear side of a vehicle. The method includes acquiringa first image data set that corresponds to a first space positionedadjacent the rear side of the vehicle, and acquiring a second image dataset that corresponds to a second space positioned adjacent the rear sideof the vehicle. The method also includes mapping the first image dataset to a first pixel array, mapping the second image data set to asecond pixel array, and calculating a correspondence between the firstpixel array and the second pixel array to produce a third image data setthat includes a first zone, a second zone, and a third zone. The methodfurther includes calculating a driving corridor that corresponds to apredicted path of motion of the vehicle, initially attempting todetermine the presence of an object in the first zone, attempting todetermine the presence of an object in one of the second zone and thethird zone if no object was detected in the first zone, determining apotential collision between the object with the vehicle corridor, andcalculating a time to collision.

Other aspects of the invention will become apparent by consideration ofthe detailed description and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the mounting positions and fields of view of twocameras on the back of a truck.

FIG. 2 illustrates the mounting positions and fields of view of twocameras on the back of a car.

FIG. 3 is a diagram illustrating the spatial imaging coverage of each oftwo cameras mounted on a vehicle and an electronic control unit thatprocesses the images obtained by the two cameras.

FIG. 4 illustrates a human-machine interface including a viewing screenthat displays images obtained by the two cameras of FIG. 1 or FIG. 2 toa vehicle operator.

FIG. 5 illustrates the logic performed by the rear-view camera system ofFIGS. 1 and 2.

FIG. 6 is a continuation of the logic of FIG. 5.

FIG. 7 illustrates the digital mapping performed by the logic of FIGS. 5and 6.

FIG. 8 illustrates the logic used to improve the processing speed of thelogic of FIGS. 5 and 6.

DETAILED DESCRIPTION

Before any embodiments of the invention are explained in detail, it isto be understood that the invention is not limited in its application tothe details of construction and the arrangement of components set forthin the following description or illustrated in the following drawings.The invention is capable of other embodiments and of being practiced orof being carried out in various ways.

A rear-view camera system 20 provides multiple functionalities with theuse of two cameras 22, 24 mounted on the rear side 26 of a vehicle 28.It is to be understood that the cameras 22, 24 may be mounted on othersuitable surfaces of the vehicle 28 (e.g., rear, side, top, orunderside), whereby the cameras are able to monitor the space adjacentthe rear side 26 of the vehicle. The rear-view camera system 20 capturesdigitized images from a space positioned adjacent the rear side 26 ofthe vehicle 28. Image data from the cameras 22, 24 is provided to anelectronic processing unit 29 (FIG. 3), also referred to as anelectronic control unit (“ECU”) which may include a programmablemicroprocessor, memory, operating system software, and other relatedcomponents. Conditioning circuits or peripheral drivers (“C/D”) 30 and31 may be used to interface the electronic processing unit 29 with thecameras 22, 24.

FIG. 1 illustrates a vehicle 28 in the form of a truck 28A that includesa rear-view camera system 20A. The rear-view camera system 20A includesa first camera 22A (herein referred to as a right camera) mounted near aright side 30A of the truck 28A and a second camera 24A (herein referredto as a left camera) mounted near a left side 32A of the truck 28A. Theright and left cameras 22A, 24A are similar and operate in a similarmanner. The right and left cameras 22A, 24A both have a field of view(“FOV”) of approximately 100 degrees. The right camera 22A is mountednear a top surface 34A of the truck 28A above a right taillight 36A, andthe left camera 24A is mounted near the top surface 34A of the truck 28Aabove a left taillight 38A. The right camera 22A is directed toward theleft side 32A of the truck 28A such that the right camera 22A acquiresimage data corresponding to the space positioned immediately behind thetruck 28A toward the left side 32A. The left camera 24A is directedtoward the right side 30A of the truck 28A such that the left camera 24Aacquires image data corresponding to the space positioned immediatelybehind the truck toward the right side 30A.

In other constructions, the right camera 22A can be different (e.g.,have a different FOV) than the left camera 24A. The FOVs of the camerastypically range from 90 degrees to 130 degrees. In some constructions,the cameras 22A and 24A are mounted in different positions on the truck28A, or the cameras 22A and 24A are aimed in different directions.

The truck 28A also includes a pair of front wheels 40A and a pair ofrear wheels 42A. During operation of the truck 28A, an operator steersthe front wheels 40A to direct the truck 28A in a desired direction. Asis known, turning can occur regardless of whether the vehicle is movingforward (a forward mode) or in reverse (a reverse mode).

FIG. 2 illustrates a vehicle 28 in the form of a car 28B that includes arear-view camera system 20B. The rear-view camera system 20B is similarto the rear-view camera system 20A illustrated in FIG. 1. Likecomponents have been given like reference numerals and will be discussedbriefly. The rear-view camera system 20B includes a right camera 22Bmounted near a right side 30B of the car 28B and a left camera 24Bmounted near a left side 32B of the car 28B. The left and right cameras22B and 24B are similar and operate in a similar manner. The right andleft cameras 22B and 24B each have a FOV of approximately 100 degrees.The right camera 22B is positioned adjacent a license plate 36B and isdirected toward the left side 32B of the car 28B such that the rightcamera 22B acquires image data corresponding to the space positionedimmediately behind the car 28B toward the left side 32B. Similarly, theleft camera 24B is positioned adjacent the license plate 36B and isdirected toward the right side 30B of the car 28B such that the leftcamera 24B acquires image data corresponding to the space positionedimmediately behind the car 28B toward the right side 30B.

In some embodiments, including those illustrated in FIGS. 1 and 2, thecameras used are CMOS- or CCD-type cameras and capture image streamscontinuously in the visible spectrum, the infrared spectrum, the nearinfrared (“NIR”) spectrum, or a combination. In one embodiment, thecameras are sensitive to light having a wavelength in the range of about400 nm to about 1000 nm, and an IR filter is used on the cameras toblock wavelengths above about 1200 nm. In some embodiments, the vehicleor the cameras are equipped with NIR light sources 23 (e.g., LEDs) toprovide illumination of areas of interest. Equipping the vehicle,cameras, or both with NIR light sources helps improve the quality ofimages obtained by the cameras, particularly at night.

FIG. 3 schematically illustrates the space adjacent the rear side 26 ofthe vehicle 28 in planar view. The right camera 22 has a FOV of 100degrees and acquires images from 0 degrees to 100 degrees, as defined inFIG. 3. The left camera 24 has a FOV of 100 degrees and is positionedsuch that the left camera 24 acquires images from 80 degrees to 180degrees, as defined in FIG. 3. The cameras 22 and 24 are positioned suchthat their respective FOVs overlap and the cameras 22 and 24 have acombined FOV of 180 degrees. In the embodiment shown, the imaged spaceis defined by five zones. Zone 1 is positioned directly behind thevehicle 28 and includes all the space imaged by both the right camera 22and the left camera 24. Zone 2 is positioned behind and to the left ofthe vehicle 28 and includes all the space imaged by only the rightcamera 22. Zone 3 is positioned behind and to the right of the vehicle28 and includes all the space imaged by only the left camera 24. Zone 4is positioned behind the vehicle 28 toward the left side 32 and isimaged by only the left camera 24. Zone 5 is positioned behind vehicle28 toward the right side 30 and is imaged by only the right camera 22.

FIG. 3 also illustrates an exemplary driving corridor 44. When thevehicle 28 is driven in a forward direction, the vehicle 28 moves in adirection defined by arrow 46. When the vehicle 28 is driven in areverse direction, the vehicle 28 moves in a direction opposite thearrow 46. Specifically, the front wheels 40 and the rear wheels 42travel substantially along the driving corridor 44 as the vehicle 28moves.

In operation, when the operator turns on the vehicle 28, power isprovided to the ECU 29 of the vehicle 28. As noted, the ECU 29 includeshardware and software and these components cooperate to execute thelogic of the rear-view camera system 20. As shown in FIG. 3, aninput/output (“I/O”) interface 47 obtains images from the cameras 22, 24and provides the images to the ECU 29. The I/O interface 47 may processthe images from the cameras 22, 24 before providing the images to theECU 29. Depending on the vehicle's direction of movement, the ECU 29outputs images (e.g., panoramic images produced from images obtainedfrom the cameras 22, 24) to a viewing screen 48. As shown in FIG. 4, theviewing screen 48 is part of a human-machine interface (“HMI”) 49. TheHMI 49 is positioned on a dashboard 49 b of the vehicle and providesinformation to a vehicle operator in various formats. In addition to theviewing screen 48, the HMI 49 includes a speaker 49 c for providingaudible information to the vehicle operator and one or more warninglights or light emitting diodes (“LEDs”) 51 for providing visualinformation to the vehicle operator. In some embodiments, the HMI 49also includes one or more haptic interfaces for providing tactileinformation to the vehicle operator, such as vibrations or movements inthe vehicle's steering wheel, seat, brake pedal, etc.

When the vehicle 28 is operated in the forward mode (i.e., when theoperator drives the vehicle 28 in the forward direction 46), therear-view camera system 20 provides multiple systems andfunctionalities. The rear-view camera system 20 provides rear-endcollision mitigation using video-based vehicle detection algorithms. Oneof the ways in which the consequences of a collision are mitigated is byusing the information generated by the system 20 to pre-activaterestraint systems. For example, if information from the system 20indicates that a rear-end collision is imminent, headrests are movedinto a position that better protects vehicle occupants, seat belts arepre-tensioned, air-bags readied to deploy, or a combination thereof.Information from the system 20 is also provided (in some embodiments) toan automatic emergency braking system. Automatic emergency brakingenhancement is achieved by determining the speed of the vehicle 28 aswell as the speed and following distance of a second vehicle travelingbehind the vehicle 28. If the vehicle 28 is equipped with aforward-looking camera, radar, or lidar system and that system isgenerating information that would cause deployment of automaticemergency braking due to an imminent forward collision, and the system20 is simultaneously generating information that a rear collision isimminent, information from the system 20 is provided to theforward-looking system so that the forward-looking system takes bothevents into consideration and determines a control compromise to achievean improved, overall safety result. The rear-view camera system 20 alsoprovides a lane departure warning (“LDW”) system and a lane keepingsystem (“LKS”) by detecting the lane markers behind the vehicle andfitting lane models to predict the relative vehicle position in thelane. The system can also detect other vehicles in blind spots of amirror or approaching fast from behind for lane-change assist. In otherwords, if the system is operating while the vehicle is moving forward(for example, traveling 120 kph on a highway), information generated bythe system can be used to provide blind-spot detection and lane-changeassistance.

When the vehicle 28 is in a reverse mode (i.e., when the operator drivesthe vehicle 28 in the reverse direction), the rear-view camera system 20provides multiple systems and functionalities that may be similar to ordifferent than the functionalities provided when operating the vehicle28 in the forward direction. The rear-view camera system 20 provideshigh-quality rear-view video stream with a 180 degree FOV, distortioncorrection, and obstacle highlighting, as explained in more detailbelow. The rear-view camera system 20 provides a back-over warningsystem with audible, visible, and haptic feedback. Some examples ofaudible feedback include beeps and voice alerts. Examples of visiblefeedback include light emitting diodes (“LEDs”) on the dashboard of thevehicle and object highlighting. Examples of haptic feedback includebrake vibration, steering wheel vibration, seat vibration, safety beltvibration, brake jerk, etc. The ECU may also actively control the braketo partially or fully stop or steer the vehicle to avoid collision. Therear-view camera system also provides brake pre-charge, a back-overavoidance system with active braking, video-based obstacle detection, anautomatic parking system that accurately detects open space andobstacles, and a cross traffic alert system that detects objects with acrossing motion.

FIGS. 5-8 illustrate the operation of the rear-view camera system 20after the vehicle 28 is turned on. At step 52, the right and leftcameras 22 and 24 acquire image data corresponding to one frame orimage. The right and left cameras 22 and 24 are video cameras andacquire data in a substantially continuous manner. For simplicity, oneiteration through the logic of FIGS. 5-8 will be explained in detail. Itis to be understood that the logic is executed rapidly by the ECU 29 andthe data is acquired and displayed in a substantially continuous manner.

As illustrated at step 52, the right camera 22 acquires a frame of imagedata in the form of a right image data array, and the left camera 24acquires a frame of data in the form of a left image data array. Eachimage data array is an array of pixels that corresponds to therespectively imaged spaces. At step 53, the image correspondence iscalculated between the right image data array and the left image dataarray. The right and left image data arrays are acquired synchronously(i.e., at substantially the same time) and thus include pixels in Zone 1that correspond to the same three-dimensional points in space. Becausethe cameras 22 and 24 are positioned on opposite sides of the vehicle28, the space corresponding to Zone 1 is imaged from differentperspectives. The correspondence calculation determines which pixels inthe right data array and in the left data array correspond to the samepoint in three-dimensional space. At step 54, the result of thecorrespondence calculation is used to determine a depth map for Zone 1.More specifically, the depth map is calculated in Zone 1 from adisparity calculation.

The ground plane (i.e., the set of data that corresponds to the surfacethat supports the vehicle 28) is assumed to be present in every imagedata array and is, therefore, removed from the depth map at step 56.Thus, only objects positioned above the ground plane remain in the depthmap. A segmentation is performed on the depth map at step 58. From thesegmented data, object recognition is performed (step 60) to determinethe presence of any objects of interest positioned in Zone 1 (i.e., inthe depth map) using extract descriptors. The extract descriptorsinclude image features (e.g., intensity, color, texture, etc.),three-dimensional shape information, and others. Performing obstaclerecognition using the extract descriptors allows further classificationof the objects of interest. Thus, false positives are reduced, andobjects can be identified in Zone 1 (step 62).

As shown in FIG. 3, each Zone may include an object 63, such as ashopping cart or child. The ECU 29 uses extract descriptors to identifythese objects 63 within a particular zone. When an object is present inthe depth map, the vehicle data is acquired (step 64). Vehicle data caninclude a velocity, acceleration, steering angle, etc. The vehiclecorridor is predicted based on the steering angle and the location ofthe centroid of the vehicle. The vehicle corridor has a widthapproximately equal to the width of the vehicle. The predicted vehiclecorridor has a polynomial curve that depends on the steering angle ofthe vehicle and the vehicle dynamics particular to the car (such aswheel base, weight, velocity, and acceleration). The vehicle corridorchanges over time (assuming that the vehicle is moving).

The objects 63 identified in step 60 are tracked and equations are usedto determine whether the tracked object can collide with the drivingcorridor. Specifically, it is determined whether the tracked object ison a collision path with the driving corridor at step 70.

If the vehicle is in reverse mode, (determined at step 72), the ECU 29produces a panoramic image (e.g., an approximately 180 degree image)from the image data acquired by the right and left cameras 22 and 24 atstep 74. The ECU 29 produces a panoramic image using image stitching.Image stitching combines or “stitches” together multiple images withoverlapping fields of view to produce a single panoramic image. Imagestitching includes multiple steps or processes, such as calibrating theindividual images, registering the images using direct or feature-basedimage alignment methods to determine image orientation and overlaps, andblending the multiple images to combine overlapping sections and correcterrors or distortions. The panoramic image can also be generated in amanner that provides a different perspective (e.g., an aerial view asopposed to an elevational view) by performing image transformation (tochange the perceived viewing angle) followed by stitching of thetransformed images. In such an embodiment, the system also includes aninput or selection mechanism (e.g., an actual or virtual button in thecockpit of the vehicle) so that the driver can select or choose whichperspective or point of view of the panoramic image is displayed to thedriver.

After generating the panoramic image at step 74, the ECU 29 highlightsany tracked objects in the panoramic image at step 76 before displayingthe panoramic image to the operator at step 78. As shown in FIG. 4, theECU 29 displays the panoramic image to the operator on the viewingscreen 48 of the HMI 49.

Regardless of whether the vehicle is in reverse mode or forward mode, awarning is produced (step 80) through the HMI 49 when an object is on acollision path with the driving corridor. The warning is produced byhighlighting, marking, or flashing tracked objects in a panoramic imagedisplayed on the viewing screen 48, illuminating one of the warninglights 51, outputting an audible alert with the speaker 49 c, and/orgenerating a tactile warning with a haptic interface (see FIG. 4). Ifthe operator does not react or fails to respond to the warning, thesystem reacts or responds by calculating the time to collision (TTC) atstep 86. The TTC is compared to a predefined threshold. If the TTC isbelow the threshold and the vehicle is moving in reverse, the systemdirects active braking to avoid a collision (step 90). In other words,the rear view camera system generates a braking command, and the brakingsystem so informed or activated forms an active braking system. When acollision is unavoidable, additional safety features are activated(e.g., seatbelt, headrest, etc.) during collision mitigation (step 88).These additional components, particularly when activated or informed bythe rear-view camera system, form a collision mitigation system. Thecollision mitigation system receives data such as object type, collisionspeed, collision position, etc. and reacts accordingly by sendingsignals to safety devices such as seatbelts, air bags, active headrests, and the like.

If no object is detected in Zone 1 (step 62), then Zones 2 and 3 areprocessed. (To reduce the amount of computation, when an object isdetected in Zone 1, Zones 2 and 3 are not examined.) The previous imagedata for Zones 2 and 3, also referred to as adjacent image data, isstored in a buffer. As best seen by reference to FIG. 6, correspondenceis calculated between the current image data for Zone 2 and the previousimage data for Zone 2 (step 92). Similarly, correspondence is calculatedbetween the current image data for Zone 3 and the previous image datafor Zone 3 (step 92).

From the correspondence data, optical flow is performed to calculate amotion flow map for Zones 2 and 3 at step 94. Optical flow is performedusing the correspondence from the current and previous image data. Theoptical flow is assumed to be caused only by objects in Zones 2 and 3because the motion map is only calculated when 1) the vehicle is inreverse mode and 2) no objects are detected during step 62 of FIG. 5.(Although, the disparity calculation for Zone 1 is always active, apotential back-over collision along the driving corridor hassignificance (only) when the vehicle is in reverse mode. However,detection of an object in Zone 1 could be due to a relativelyfast-moving object approaching from behind, which could, depending onthe situation, lead to a rear-end collision or a front-end collision(e.g., if the object is approaching faster than the vehicle can driveaway.) The pixels corresponding to objects in Zones 2 and 3 are assumedto be moving in the same height to simplify the equations and becauseonly crossing motion is relevant. Thus, the motion is transformed to theplan view. In addition, the vehicle ego-motion (e.g., vehicle speed,vehicle direction, steering angle, etc.) is known and the real motion ofa pixel or object can be determined from the vehicle ego-motion and thetracked motion. Only real motion toward Zone 1 and toward the drivingcorridor is considered relevant. Other motion is ignored. The motionpixels are clustered to determine whether a potential object is movingtoward the vehicle and whether the potential object is on a collisionpath with the driving corridor (step 106). If no objects are detectedthat are on a collision path, then new right and left image data arraysare acquired (at step 50) and the process repeats.

If an object in Zone 2 or 3 is determined to be on a collision path(step 106), then the vehicle mode is determined. If the vehicle 28 is inreverse mode, (determined at step 110), a panoramic image is produced atstep 112 (e.g., using image stitching as described above for step 74),and the tracked object is highlighted or marked in the panoramic imageat step 114. The panoramic image is then displayed to the operator onthe viewing screen 48 of the HMI 49 (see FIG. 4) at step 116.

A warning is produced (step 118) through the HMI 49 (regardless of theforward or reverse mode or movement of the vehicle). The warning isproduced by highlighting, marking, or flashing tracked objects in apanoramic image displayed on the viewing screen 48, illuminating one ofthe warning lights 51, outputting an audible alert with the speaker 49c, and/or generating a tactile warning with a haptic interface (see FIG.4). If the operator does not react to the warning (step 122), the systemcalculates the time to collision (TTC) at step 128. If the TTC is belowa predefined threshold, the system directs active braking to avoid acollision (step 134). When a collision is unavoidable, additional safetyfeatures are activated (e.g., seatbelt, headrest, etc.) during collisionmitigation (step 132). As noted above, collision mitigation is based ondata such as object type, collision speed, collision position, etc.

Due to the complexity of the correspondence calculations performed atstep 52 of FIG. 5 and step 92 of FIG. 6, the image data is digitallymapped before being processed. FIG. 7 visually illustrates the digitalmapping performed on the image data arrays. The cameras 22 and 24 eachhave a different perspective view, which results in zones that are notstructured in a uniform manner. Thus, digital mapping is performed onthe image data to transform the image data into a more structuredformat. Digital mapping describes image transformation such asstretching or compressing some areas of an image to normalize the use ofpixel data. Some examples of digital mapping are image distortioncorrection, perspective transformation, and projection. Certain areas inthe image data may be considered more important or more useful thanother areas. The important areas can be modified to increase thedigitized pixel resolution, and the less important areas can be modifiedto decrease the pixel resolution. Digital mapping in the plan view alsoallows the non-uniformity issue to be solved. The non-uniformity issueresults from a camera view with a pitch angle that results in too manypixels in the near range. After digital mapping, the imaged space isperceived uniformly. As illustrated in FIG. 7, the image data isrestructured into data blocks to allow a faster correspondencecalculation.

As illustrated in FIG. 1, the right camera 22 is mounted on the rightside 30 of the vehicle 28 and is aimed toward the left side 32 of thevehicle 28. The image data acquired from the right camera 22 isschematically illustrated in FIG. 7 as an array 136. The image data inarray 136 corresponds to the left side 32 and includes L(Zone 1), L(Zone2), and Zone 5. Similarly, the image data acquired from the left camera24 is schematically illustrated as an array 138. The image data in array138 corresponds to the right side 30 and includes R(Zone 1), R(Zone 3),and Zone 4. Digital mapping is performed to evenly distribute the dataof the arrays 136 and 138 into corresponding sections or data blocks sothe correspondence calculations may be performed quicker. Morespecifically, the image data corresponding to Zone 1 in array 136 ismapped to Zone 1 in array 140, the image data corresponding to Zone 2 inarray 136 is mapped to Zone 2 in array 140, and the image datacorresponding to Zone 5 in array 136 is ignored. The right camera 22 hasa field of view that does not include the image data corresponding toZone 3. Thus, the image data corresponding to Zone 3 from the previousdata set is added to array 140 as the data for Zone 3. A similar digitalmapping is performed for array 138. The mapped array 142 includes themapped data corresponding to Zone 1 and Zone 3 of array 138. The imagedata in Zone 4 of array 138 is ignored. The left camera 24 has a fieldof view that does not include the data corresponding to Zone 2. Thus,the previous data set for Zone 2 is added to array 142 as the data forZone 2. Each array 140 and 142 is stored in a different data buffer.

FIG. 8 illustrates the logic for the hardware-accelerated,correspondence calculation 144. The logic includes digital mapping anddata alignment (steps 150 and 152), as discussed with respect to FIG. 7.Data alignment is the restructuring of the image data arrays intocorresponding data blocks, which are used in each correspondencecalculation. Correspondence calculations are used to calculate the depthmap and optical flow using the information in the remapped arrays 140and 142. As illustrated, the previous data for Zone 2 (step 152) as wellas the current data for Zone 2 (step 156) is used in the correspondencecalculation for Zone 2 (step 158). The data corresponding to Zone 1 fromeach array 140 and 142 (steps 160 and 162) is used for thecorrespondence calculation of Zone 1 (step 164). The data correspondingto the previous data for Zone 3 (step 168) and the current data for Zone3 (step 166) is used in the correspondence calculation for Zone 3 (step172).

Correspondence calculations are large and take a long time to computewhen high accuracy and robustness are desired. Thus, to furtheraccelerate the speed of the correspondence calculations, thecorrespondence calculations are performed in parallel using a speciallydesigned structured element machine in computational devices or aparallel hardware architecture such as field programmable gate arrays(“FPGAs”) and application specific integrated circuits (“ASICs”). Ofcourse, other hardware can be used to accelerate the calculations.

Thus, the invention provides, among other things, a multi-functionalrear-view camera system that provides a variety of collision mitigationand avoidance systems. Various features and advantages of the inventionare set forth in the following claims.

1. A rear-view camera system for a vehicle, the rear-view camera systemcomprising: a first camera positioned on a rear side of the vehicle, thefirst camera producing a first data set that includes image datacorresponding to a first field of view; a second camera positioned onthe rear side of the vehicle spaced a horizontal distance from the firstcamera, the second camera producing a second data set that includesimage data corresponding to a second field of view, the first field ofview and the second field of view covering a space adjacent the rearside of the vehicle; an electronic control unit that receives the firstdata set and the second data set and processes the first data set andthe second data set to determine a presence of an object in the spaceadjacent the rear side of the vehicle, the electronic control unitconfigured to stitch the first data set and the second data set toproduce a panoramic image that corresponds to the space adjacent therear side of the vehicle; and a human-machine interface that includes aviewing screen positioned adjacent an operator of the vehicle, thehuman-machine interface configured to receive the panoramic image anddisplay the panoramic image on the viewing screen.
 2. The rear-viewcamera system of claim 1, wherein the electronic control unit calculatesa driving corridor, determines a time to collision of the object withthe vehicle within the driving corridor, and provides a warning to theoperator of the vehicle.
 3. The rear-view camera system of claim 2,wherein the warning is one of an audible warning, a visual warning, anda haptic warning.
 4. The rear-view camera system of claim 2, wherein thewarning includes highlighting the object in the panoramic image.
 5. Therear-view camera system of claim 1, wherein the space adjacent the rearside of the vehicle defines a field of view of between 100 degrees and180 degrees.
 6. The rear-view camera system of claim 1, wherein thespace adjacent the rear side of the vehicle defines a field of view of180 degrees or more.
 7. The rear-view camera system of claim 1, whereinthe image data is a video stream of image data.
 8. The rear-view camerasystem of claim 1, further comprising hardware configured for parallelprocessing of the image data.
 9. The rear-view camera system of claim 8,wherein the hardware includes one of an application specific integratedcircuit, and a field programmable gate array.
 10. The rear-view camerasystem of claim 2, further comprising a collision mitigation system anda braking system, the collision mitigation system and the braking systemconfigured to operate in response to determining a failure of theoperator to respond to the warning.
 11. The rear-view camera system ofclaim 1, wherein the electronic control unit calculates a drivingcorridor, determines a time to collision of the object with the vehiclewithin the driving corridor, and information generated by the electroniccontrol unit is used to provide at least one of a blind-spot warning anda lane-change indicator.
 12. The rear-view camera system of claim 1,further comprising a braking system and a forward-looking collisiondetection system, wherein imminent rear-collision information from theelectronic control unit is provided to the forward-looking collisiondetection system, and wherein the forward-looking collision detectionsystem is configured to control the braking system based on both theimminent rear-collision information and detecting an imminent forwardcollision.
 13. The rear-view camera system of claim 1, wherein theelectronic control unit changes a viewing angle of the panoramic imagein response to user input.
 14. The rear-view camera system of claim 1,wherein the first and second camera are operable to capture imagestreams in the visible spectrum and the infrared spectrum.
 15. Therear-view camera system of claim 1, further comprising infrared lightsources operable to illuminate the first field of view and the secondfield of view with infrared light.
 16. A method of detecting an objectadjacent a rear side of a vehicle, the method comprising: acquiring,with a first camera, a first image data set that corresponds to a firstspace positioned adjacent the rear side of the vehicle; acquiring, witha second camera, a second image data set that corresponds to a secondspace positioned adjacent the rear side of the vehicle; in an electronicprocessing unit, mapping the first image data set to a first pixelarray; mapping the second image data set to a second pixel array;calculating a correspondence between the first pixel array and thesecond pixel array to produce a third image data set; calculating adriving corridor that corresponds to a predicted path of motion of thevehicle; determining a presence of the object in the third image dataset; determining a potential collision between the object and thedriving corridor; and calculating a time to collision.
 17. The method ofclaim 16, wherein the mapping of the first image data set and themapping of the second image data set is performed by one of anapplication specific integrated circuit and a field programmable gatearray.
 18. The method of claim 16, further comprising determining if thevehicle is operating in a reverse mode, and, if the vehicle is operatingin the reverse mode, producing a panoramic image corresponding to thefirst space and the second space and displaying the panoramic image toan operator of the vehicle.
 19. The method of claim 18, furthercomprising highlighting the object in the panoramic image.
 20. Themethod of claim 18, changing a viewing angle of the panoramic image inresponse to user input.
 21. The method of claim 16, wherein the thirdimage data set includes image data that corresponds to a spacepositioned directly behind the rear side of the vehicle.
 22. The methodof claim 21, further comprising calculating a depth map for the imagedata that corresponds to the space positioned directly behind the rearside of the vehicle, and calculating a motion map for adjacent imagedata of the third image data set, the adjacent image data correspondingto space adjacent to the space positioned directly behind the rear sideof the vehicle.
 23. The method of claim 16, further comprising warningan operator of the vehicle when the potential collision is determined.24. The method of claim 23, wherein warning the operator includes one ofproviding an audible warning, a visible warning, and a haptic warning.25. The method of claim 23, further comprising detecting a failure ofthe operator to respond to the warning and actively braking to attemptto avoid the collision in response to the failure.
 26. The method ofclaim 23, further comprising detecting a failure of the operator torespond to the warning, comparing the time to collision to a threshold,and providing at least one of collision mitigation and active brakingwhen the time to collision is less than the threshold.
 27. The method ofclaim 16, providing at least one of a blind-spot warning and alane-change indicator based on information generated by the electronicprocessing unit.
 28. The method of claim 16, further comprisingproviding imminent rear-collision information from the electronicprocessing unit to a forward-looking collision detection system based onthe potential collision and the time to collision, detecting an imminentforward collision, providing control signals from the forward-lookingcollision detection system to a braking system based on both theimminent rear-collision information and detecting the imminent forwardcollision.
 29. The method claim 16, wherein acquiring the first imagedata set and the second image data set includes capturing image streamsin the visible spectrum and the infrared spectrum.
 30. The method ofclaim 16, further comprising illuminating the first field of view andthe second field of view with infrared light.